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ABSTRACT 


UNIPARSE: A Universal Parser, is a FORTRAN program implemented on 
an IBM 360/67. It is capable of parsing any finite, recursively enumerable 
language. The system simulates a Universal Turing Machine and utilizes 


bounded Breadth-First search and development techniques to evaluate 


Type O languages. 
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I. INTRODUCTION 


ine Obyective Of “tlis paper is to6 present a generalized metioas1on 
parsing finite, recursively enumerable languages. The system developed 
to implement this method, UNIPARSE, will parse any language of the above 
type given the grammar for that language in canonical form. 

The motivations for attempting to develop such a system were twofold. 
First, parsing algorithms for all languages are dependent upon the syn- 
tactical constructs of the language. Since recursively enumerable 
languages represent one of the largest classes of languages, the develop- 
ment of a system capable of parsing languages belonging to this class 
provides for the development and implementation of computer languages of 
a less restrictive nature than those in existence today. 

Second, although there do exist generalized methods for parsing computer 
languages of certain types (Simple Precedence and SLR languages, for 
example), there is no existing system which can parse all computer lan- 
guages. UNIPARSE provides such a mechanism. 

The system was as for use in a batch processing environment but 
could eas ily be modified for an interactive system. A Users Manual 
(Appendix A) and a documented program of the system (Appendix B) are 
provided as a part of the presentation. Appendix C contains examples 


of the parsing of languages by the system. 





Pi ovolenl DESCRIPTION 


UNIPARSE is a simulation of a nondeterministic Universal Turing 
Machine. The system, given the grammar for a recursively enumerable 
language, will construct a Turing Machine capable of recognizing words 
from the language. Furthermore, if a string of symbols is input to the 
system, UNIPARSE simulates the operation of the constructed Turing Machine 
and determines whether or not the input string is a member of the language. 

The system can conceptually be divided into two major subsystems. The 
first of these, titled CONVERT, when given a language grammar constructs 
a Turing Machine capable of recognizing members of the language. The 
second subsystem, called PARSER, takes as inputs the Turing Machine con- 
structed by the first subsystem and a string of symbols. It then proceeds 
to simulate the operation of the Turing Machine, and, by doing so, parses 
the input string. Both subsystems are described in detail in the following 


sections, The first subsystem discussed is CONVERT. 


Eee CONVERT 
This portion of the system converts the set of productions for a gram- 
mar into a Turing Machine which recognizes the language generated by the 
grammar. The following items are inputs to this subsystem: 
1. The set of productions which define the language grammar. 


These productions are input in the following format: 
(left™ bast) ~ Sa(kiene Pare) 


where "Lett Part" and “Right Part’ consist of Stringsme- 


a "1 


symbols from the alphabet of the language and : is 
interpreted to mean “is replaced by." 
2. The set of terminals for the language. 
CONVERT applies the above inputs and transforms them into rules for 


a Turing Machine. These rules take one of the following forms: 
(Qy 5 Seo 0 esa = (Q., Wa o+* Yap Left) 


(Qi; XaXQee-eX) = (Q; 5 Yo +Ym? Right) 


(Gi, 1h) ---,) = (QF eyo eee 


where 


1. Q. is the present state of the Turing Machine. 
i 


De XyXo+6-X, are symbols on the tape of the machine which are 
currently being scanned. 

3 Q. is the next state the Turing Machine will assume. 

4. YyYor++¥, are symbols which will replace the symbols 


XpXoee+X, ON the tape of the Turing Machine. In accordance 


with the definition of recursively enumerable languages, ‘“m" 


can be less than, equal to, or greater than "n", 


>. Left and Right indicate the direction in which the tape head 
will next move. Null indicates an epsilon move. 
The constructed rules are formed by applyiug one or more elements from 


the set of fundamentals Listed in Table l. 


tt tt 1t 


In Table 1, is the start state for the Turing Machine. 


“QQ Q05 


tt 


is the final (accepting) state. "Q and are 


a 
terminal Q terminal 
States which identify with particular terminals in the vocabulary of the 


fanguage. All other states are intermediate states. "'@yye'"!" and aaa. 


are delimiting and replacement symbols used internally by the system. 





HO; 
ibe 
12, 
13. 
14. 
5% 
For 


lke 


ryt 


itself a left part of some production. 


(Qog> Left Part) e 
(Qgg: Terminal) - 
Cie _ 
(Qoq2 +) 
(Qog> © : 
(Qo,» Terminal) = 
Qo» ® : 
(Qg3» Terminal) E: 
(Qy3, #) : 
(293 1) a 
Qi # : 
(Q,> @ ' 
(Qoy4> 1) : 
(Qyeee rep mlcemunel) = 
Saree @) 7 
feat = 
(Q, ceminal®» Terminal) = 
Table 1. 


one, 
Qoos 
Qoqs 
Gre 
(Qos 
(or? 
Gre 
(Q 
(Qq35 
Gane 
Qo, 
(264; 
Gay 
(Q 
(Q 


terminal 


. $ 
terminal 


° 3 
terminal 


Right Part, Nuis 
Terminal, Right) 


V*, Right) 


YS het) 


@, Right) 


Terminal, Left) 


@, Right) 

#, Left) 

#, Right) 

|, Here) 

#, Left) 

@, Left) 

L, ReohE) 

Terminal, Left) 
@, Left) 


<, ee Riche) 


terminal 


(ae 


!, Right) 


Ebastwort Fundamentals 


is a symbol in the imput alphabet which contributes to, but is not 


The symbols Bo and Caen 


the third example shown in Appendix C represent this type of symbol. 


Pitett art. 


and 


Ra oie are 


are aS “detined carluer, 


The rationale 


behind the use of each of the fundamentals listed in Table 1 is explained 


in the description of the second subsystem, PARSER. 





B. PARSER 
Once the rules for a particular Turing Machine have been constructed 
UNIPARSE takes the following as inputs: 
1. <A string of symbols which is to be parsed. 
2. A number which represents the maximum length of the left 
side of any production in the input grammar. 
To the input string is addended the following symbols: 
1, "!" dis addended to the left side of the string. 
2. “@" is addended to the right side of the string. 
3. The start symbol for the grammar is addended to the right 
of the "@' symbol. 
4, “wis addended to the right of the start symbol. 


it. for example, " is the inputted string and "S" is the 


t 
ca ese ad °Sy 
Start symbol, the final results of the above operations will be the 


following string: 
1 S)S5.-.-S,@ S ! 


The resultant string is now placed on the tape of the Turing Machine, 
the tape head is positioned so that it is scanning the start symbol, and 
the machine is placed in its initial state. This portion of the system 
then commences to simulate the operation of the constructed Turing Machine. 
It does so in the following manner: 

1. The string of symbols which lie between the "@" symbol 


try tt 


and the rightmost symbol are scanned to determine if 
any substring consists of the left part of any production 
rule. The tape head scans substrings of length equal to or 


less than the maximum length of the left side of any production 


in the input grammar. If a left part is discovered, rules 
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constructed from Fundamental #1 in Table 1 are applied and 
the left part is replaced by the appropriate right part. 

The resultant string is then stored. 

If at anytime during this sequence of scanning, the tape 
head scans a terminal or a non-terminal which is not a left 
part, these symbols are replaced by themselves in accordance 
with Fundamentals #2 and #3. 

If the tape head scans the rightmost "!" symbol while in 
the initial state, Fundamental #4 is applied and the tape 
head moves left to determine if a string of terminals have 
been generated. Rules constructed from Fundamental #6 allow 
for Mtikbis. 

If a string of terminals has been generated, the tape head 
will eventually arrive at the "@" symbol while still in 


state "Qo1 . Fundamental #7 will then be applied, and the 
machine goes into state "Q03.° 

From this point on the machine will scan alternately to the 
right and to the left of the "@" symbol to determine if 

the generated string is identical to the input string. Rules 
constructed from the remaining fundamentals implement this. 
If, during the scanning process, a symbol is scanned for 
which none of the rules for the Turing Machine apply, the 
string containing that symbol is removed and development of 
other strings from this string is abandoned. 

Once this occurs, PARSER removes the earliest previously 
derived string from storage and repeats steps l through One 


after abandoning a particular path of derivation, PARSER finds 


that there are ne remaining Strings in storage, the spur 


String is rejected. 





The method ees above for constructing the derivation tree for a 
language is analogous to a Breadth~First tree search. Figure 1 illustrates 
this type of tree development for the string "12" which is a member of the 
language consisting of all words containing an equal number of 1's and 2's., 
The corresponding rules which define the grammar for this language are 
included in the lower right side of the figure. Each node in the deriva- 
tion tree represents a derived string. Each connecting arc corresponds 
to the applicable rule which results in the derivation. The nodes are 
numbered in terms of their sequence of development. 

PARSER, in this case, initially places the start symbol, "S”; on 
the Turing Machine, together with the bracketing symbols and the input 
string. The Turing Machine scans the start symbol and replaces it with 
the string "2A". PARSER stores this string and the Turing Machine looks 
for other deivations from the start symbol. It finds "1B" and replaces 
the start symbol with it. PARSER then stores this string and the Turing 
Machine again looks for other derivations. Finding none, the earliest 
derived string, "2A", is removed from storage and placed on the tape. 
Derivations numbered four and five are then developed and placed in 
storage. The Turing Machine discovers that derivation number six results 
tea string of terminals. At this point it compares the derived string 
with the input string. The two are not the same, and the derivation is 
rejected. There are no further derivations from the string "2A" so 
"1B'"" is removed from storage and the Turing Machine commences to develop 
it. As we see from Figure 1. The string "12" is eventually derived, 
compared and accepted at Node 14. The rules for a Turing Machine for 
this language are provided, together with the full derivation of a string 


from the language, in example 1 of Appendix C. 
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The reason UNIPARSE develops derivation trees for languages ina 
Breadth-First fashion rather than in some other way is linked to a well- 
known phenomena in Automata Theory known as the "Halting Problem." 

Simply stated, the "Halting Problem" is as follows: Given any Turing 
Machine with some arbitrary string of symbols on its tape, there is no 
guarantee that the machine will eventually halt either accepting the 
string as a member of a language or rejecting it as being invalid. An 
informal proof of this theorem is found in Reference 1. It is sufficient 
to say here that the problem is caused by some series of rules for a non- 
deterministic Turing Machine which, beginning with the replacement of 
some symbol on its tape, eventually results in a string containing that 
symbol. 

To illustrate the "Halting Problem" and what might happen had we not 
developed the derivation tree in Figure 1 ina Breadth-First fashion, 
let us suppose we had developed it using some Depth-First technique. This 
type of development chooses some single path and pursues this path until 
it is not possible to generate further strings. Then, and only then, will 
it attempt to derive strings lying along some other path. For example, 

a Depth-First development of the derivation tree in Figure 1 might have 
proceeded down the leftmost branch of the tree. That is, from node number 
1 to node number 2 and then to node number 4. At this point, we have 
created a recursive relationship between rule number one and rule number 
three in the grammar. Hereafter, the start symbol is always replaced by 
the sub-string "2A" and the symbol "A" in the resultant string is, in 
furn, replaced by the start symbol. It is™evident "that continues 
this path will never lead us to a string of symbols composed entirely of 


terminals. Hence, the machine will never halt. Furthermore, although the 
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input string was a word in the language, development of the tree in the 
manner just described would never have revealed this fact. 

On the other hand, construction of the derivation tree using Breadth- 
First methods ensures that the Turing Machine will not proceed down some 
single, unproductive path to the exclusion of all other paths. Hence, if 
the input string is a word in the language, we can be assured that it 
will eventually be recognized by UNIPARSE, 

Searching in Breadth, however, resolves only half of the "Halting 
Problem": that of recognizing valid strings in a language. We still 
cannot be sure that invalid strings will be rejected by the system. Even 
with a Breadth-First approach, if there exists a path in the derivation 
tree which does not end in a string of terminals, the Turing Machine will 
never halt, rejecting an invalid string. The heuristic applied by 
UNIPARSE to resolve this portion of the problem is to bound the develop- 
ment of the derivation tree. That is, the tree will be allowed to develop 
only to a certain, specified depth. 

This is a viable solution if we accept the condition that any language 
which we might wish to parse contains only a finite number of words. If 
the language is finite, and if the derivation tree for the language is 
developed in Breadth, after some finite number of derivations, all words 
in the language will be generated. The bound for the tree is set at the 
depth at which this occurs. If the input string has not been generated 
prior to reaching this level, it cannot be a member of the language and 
will be rejected. The system does not determine for arbitrary grammars 
what this level is. It must be specified by the user for the language 


he wishes to parse. 


186) 





C. SUMMARY 

UNIPARSE is a simulation of a Universal Turing Machine; which, given 
specified inputs, will parse any finite, recursively enumerable language. 
The system consists of two major subsystems: CONVERT, which constructs 
a Turing Machine for parsing the language; and PARSER, which uses the 
constructed machine and a Bounded Breadth-First derivation tree develop- 


ment scheme to parse input languages. 
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IIIT. CONCLUSION 


It is clear from the preceeding description that UNIPARSE is capable 
of parsing finite, recursively enumerable languages. The system has been 
implemented and tested on Regular, Context Free, Context Sensitive, and 
Type O languages. The following are the observed results from these 
EeSts: 

1. The system, in fact, defines a workable method for parsing 
recursively enumerable languages. UNIPARSE parsed correctly 
all test languages. Appendix C contains examples of some of 
these tests. 

2. The system as presently developed is somewhat slow and 
demanding of storage space. This is due, in part, to the 
fact that the system was designed to demonstrate the feasi- 
bility of implementing a generalized method for parsing 
languages. As a result, programming efficiency was sacrificed 
for readability and clarity of understanding. 

The value of the system lies in its generality. The existence of 
UNIPARSE allows for the development of computer programming languages of 
higher order than those presently available. In addition, the system pro- 
vides a single method for parsing all computer languages of order Type O 


@r Less, 


Tis) 





APPENDIX A 


USER's MANUAL FOR UNIPARSE 


The system is designed for batch processing. The user, in addition 


to the main program, must input the following items: 


The 


le 


Item: The set of productions for the grammar of the language 
to be parsed. 

Item: The set of terminals for the language. 

Format: 80A1 

Item: The Start symbol for the language. 

Format: 1A4 

een: ue een length of the left side of any production 
in the input grammar. 

Format: 1120 

Item: The number or nodes to be generated by the system. 


Format: 116 


following are outputs from the system: 


wae 


The input set wi productions. 

The set of terminals for the language. 

A listing of the rules for the Turing Machine used to recognize 
the input language. 

The start and final (accepting) states of the Turing Machine. 

A sequential listing of the strings of symbols generated by 


the system. 
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6. The location of the symbol on the tape of the Turing Machine 
which is currently being scanned. 
7a) the state which the Turing Machine ic yin, 

In addition to the above outputs, UNIPARSE provides the following 
Outputs which are dependent upon whether or not the input string is a 
valid word in the language or not: 

1. If the word is rejected, the system informs the user of this 
and points out the invalid symbol. 
2. If the input string is a word in the language, the system 
Pout Nkorms suet user. 

The following restrictions are programmed into the system: 

1, Languages containing as members of their alphabets the 
following symbols are not allowable: 


a. ss Hie 


jeoyeee ie 
Cit en 
These symbols are used ‘internally by the system, and, hence, 
cannot be treated as language symbols. Replacement of these 
symbols by, other symbols provides a simple solution to this 
problem. 

2. The.program currently provides for one minute of computing 
time and occupies 150k of core space. If more time or space 
are Tequired , app nepulatc mopeConemoleecards forsee wna 


program must be modified. 


Ly 
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